Images photographed on film often need retouching before they can be used in a setting requiring high quality imaging. One task addressed by such retouching is removing localized blemishes from the image. These blemishes may be part of the original image, such as undesired marks on skin, or they may have been introduced by the photographic process, such as dust or scratches. In non-electronic image manipulation, this retouching is typically performed by air-brushing.
The digital analogue of air-brushing is pixel cloning. This is an operation which is common to most digital image editing software packages. Cloning consists of copying pixels from one part, the source area, of the image to another part, the destination area, of the image. The copying takes place in a small region in the neighborhood of the cursor on the screen, under the control of the computer mouse or other pointing device. Thus the operation is completely under the interactive control of the operator, and the operator can easily change the relative offset between the source area and the destination area. Typically, as the user moves the computer mouse, the location of the source area moves along with the location of the destination area, maintaining a fixed offset between them, with pixels being copied as the operator moves the mouse. It is time-consuming partly because of the need to select source areas for the clone which closely match the color and brightness of the area surrounding the destination.
The copying may be done either to cover over some unwanted object in the destination area of the image, such as blemishes as mentioned above, or to make a copy in the destination area of an object in the source area. In general, but especially in this latter application, the source area may come from another image entirely. In preparing images for professional use, cloning consumes a large part of the time an operator spends working on an image.
Variations on cloning techniques allow for different brush shapes, i.e., the shape of the area which is copied. The shape may be predefined, for example rectangular or circular, or be user-defined. In addition, some randomness can be added to the copying operation to simulate the random nature of true air-brushing, as disclosed in commonly assigned patent application Ser. No. 746,664, now U.S. Pat. No. 5,276,510, incorporated herein by reference.
With a real air-brush, the depositing of ink occurs continuously as the operator moves the air-brush. In a digital system, the location of the mouse is only reported at discrete time intervals. Thus a high quality system will interpolate between the reported locations in order to simulate the smooth movement of the pointing device.
In U.S. Pat. No. 4,577,219, retouching of a color image by copying pixels from one area to another is disclosed. In the '219 patent, original color signals are obtained by scanning of the color image, and then converting the original color signals into color values for a variety of different color components of the color signals. However, the '219 patent only discusses copying pixels unchanged from one area to another.
A problem with cloning that has not been addressed by any existing image editing system is that the cloning operation can be very painstaking for the user. The operator must take care that the color and brightness of the pixels being copied, the source pixels, match the color and brightness of the pixels surrounding the destination area. If the new pixels differ significantly from the pixels surrounding their new location, the operator will simply have replaced the old artifact with a new one.
One could perhaps measure the average color in the surrounding area and simply paint over the artifact with that color. Unfortunately, natural imagery has a natural texture, provided by film grain if nothing else, and simply painting digitally with a constant color will not produce the same texture. Even if the texture were simulated, the destination region would still have different lighting gradients than its surrounding pixels.
In using a cloning operation, an operator will find a closely matching region as the source for the clone. This usually is not an exact match and there may well be a discernible boundary between the new pixels and the surround. The typical solution to this is to blur the region of the boundary slightly in order to mask the boundary. This achieves usable images at the cost of taking more operator time.
It is seen then that there exists a need for an improved method of adjusting the brightness and color of copied pixels so that they more nearly match the surrounding area, overcoming the problems associated with the prior art.